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(54) CNN bionic eye or other topographic sensory organs or combinations of the same 



(57) The main design components underlying the 
implementation of physiologically faithful retina and oth- 
er topographic sensory organ models on CNN universal 
chips is discussed. If the various retinas are implement- 
ed on a CNN universal chip, in a programmable way, it 
can be called a "CNN bionic eye", a device capable of 
perfomning a broad range of image processing functions 
similar to those perfonned by biological retinas. The 
CNN universal machine has the special properties that 
it is 1 } programmable and 2) includes local memory. Pro- 



gramming is stored In analog and logical form (the an- 
alogic program) generated by an analogic programming 
and control unit, so the functions of the CNN universal 
machine can be modified as a function of complex in- 
ternal and external constraints. Further, several CNN bi- 
onic eyes and other topographic sensory modalities can 
be combined on a single CNN universal chip, and for, 
more complex sensory tasks, the necessary physical 
microsensons to provide the input signals can be imple- 
mented on the chip, In most instances. 
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Description 

CROSS-REFERENCE 

5 [60OI] This invention is related to issued U.S. Patent 5,1 40,670, issuedAug. 18, 1992, entitled "CELLULAR NEURAL 
NETWORK", and an earlier filed patent application entitled "CNN UNIVERSAL MACHINE AND SUPERCOMPUTER" 
having been assigned serial number 07/959,562, filed on October 13, 1992, by Tamas Roska and Leon O. Chua, both 
of which have been assigned to the same assignee as is the present patent application. 

10 FiELD OF THE INVENTION 

[0002] The present invention relates to cellular neural networks (CNN), and more particularly to afully programmable, 
or programmed, CNN supercomputer that functions as a CNN bionic eye or other topographic sensory organ, or com- 
binations of such organs whk^h may have functions that are extensions of those that occur naturally in living organisms. 

15 

BACKGROUND OF THE INVENTION 

[0003] During the past decade there has been an enomious Increase in the understanding of vertebrate retinal func- 
tion. The time and space constants have now derived for some animals, as well as the transfer functions that associate 
20 retinal elements, thus, complete dynamic models have been generated. 

[0004] A number of recent attempts have been made to implement retinal function on silicon chips and some simple 
artificial electronic realizations based on phenomenology of retinal function are available. The digital simulation of the 
tiger salamander retina model also proved to be a faithful model as well. 

[0005] A cellular neural network (CNN) is a multi-dimensional lattice array of a plurality of identical cells. In a two- 

25 dimensional planar array of cells, the array may be either square, rectangular, or hexagonal. For the n-dimensional 
case, the array in any direction can have one or more cells, i.e., there does not have to be the same number of cells 
in each of the n-dimensions of the array. With respect to any single cell, the cells closet to it are its neighbor cells. 
Additionally, the neighbor cells of each cell expand concentrically from that single cell as concentric circles, or partial 
concentric circles, of neighbor cells. Each cell in the array interacts non-linearly and in continuous time with a prescribed 

30 number of concentric circles of neighbor cells within the lattice. Adjacent cells can interact directly with each other, and 
cells not directly connected together may affect each other indirectly because of the propagation effects of the CNN. 
[0006] In a two dimensional, single-layer CNN, each circle of neighbor cells will also be two dimensional and be a 
sub-set of the cells in the single-layer of the CNN. In a three dimensional CNN, there will be at least two layers of CNNs 
and the neighbor cells may be either restricted to the single layer in which they are physically located, or the neigh- 

35 borhood of cells could also be three dimensional and incorporate portions of several layers much like layers of an 
onion. The actual application dictates whether the neighborhood of celts are two or three dimensional. 
[0007] The basic unit of a CNN is the cell. Each cell contains linear and nonlinear circuit elements, which may include 
linear capacitors, linear inductors, linear resistors, linear and nonlinear controlled sources, and Independent sources. 
[0008] The CNN universal machine and supercomputer extends the basic CNN into a cellular analog stored-program 

40 multidimensional array computer with local analog and logic memory, analogical computing units, and local and global 
communication and control units. The global analog stored-program memory concept is a key element of the CNN 
universal chip architecture providing the implementation of analogical algorithms using time-multiplex templates. 
[0009] The CNN machine is universal in the same sense as a Turing machine except that alt of the signals are analog, 
not digital. However, the so called dual computing structure in the CNN architecture allows a direct combination of 

45 analog and logic signals, without any analog-digital conversion. Recently a programmable CNN device with local mem- 
ory, the CNN universal machine was Invented with this device allowing one to program an unlimited range of spatio- 
temporal transformations, linear and nonlinear, in 2 or 3 dimensions. CNN templates have also been used to success- 
fully process pictures with areas of vastly different local illumination. 

so SUMMARY OF THE INVENTION 

[0010] The CNN universal machine and supercomputer can be used to implement any topographic bionic sensor, 
and to illustrate that, this disclosure presents the details of how to implement the CNN universal machine to build a 
bionic eye that can perfomri all the known functions of vertebrate retinas, and extend beyond the phenomenological 
55 models. The same techniques are also applicable to the implementation of other bionk: sensors using the CNN. 

[0011] The image processing functions performed by the CNN universal machine encompass, but are not limited to, 
those performed by many different vertebrate retinas ranging from salamanders to primates. All the functional properties 
of these retinas, as well as others that do not exist in nature, and other sensory organs can be implemented in the 
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same CNN universal machine. These retinas, and other sensory organs, not only respond in specific ways to the 
sensory input, but they adjust their responsive properties to match the environment and the objective of the animal. 
All of these functions and the adjustments in function performed by biological sensors are also implementabte in a 
CNN universal machine through its analogic (analog and logical) programming capability. The functional properties of 
5 the CNN universal machine are not built in (not hard wired) but can be programmed, and the programming can be self 
adapting, like the biological sensors themselves, such as a retina, to match the content and context of the visual and 
other sensory task. 

[0012] In addition, because of its unique design, the CNN universal machine can perform both sensory processing 
and adaptive functions that go far beyond vertebrate sensors, both in their complexity, range of adaptability and speed 
10 of operation. 

[0013] For example, the retina processes information sequentially across 5 sheets, or layers, of neurons, the CNN 
bionic eye can process information across a very large number of such layers In the same time period that a biological 
retina processes the same infonmation, at least 1 0,000 such "slices" or processes within that lime period. The biological 
retina adapts to a few important contexts of the visual scene, including brightness, texture and movement. The CNN 
15 universal machine bionic eye is capable of adapting to those and many more possible contexts, including those never 
even seen by biological retinas. The biological retina processes the visual scene in about 100 msec. The CNN bionic 
eye can perform these functions In less than one millionth of a second. Similar arguments are valid to othertopographic 
sensory organs (tactile, somatosensory, olfactory, etc.). 

[0014] The applications for the CNN bionic eye, or other CNN bionic sensors, are limitless. The CNN bionic eye is 
^0 an image processing supercomputer, no larger than a thumbnail, and consuming minimal power. It is capable of 
processing a visual image in one millionth of a second, then store that image, be reprogrammed in a microsecond, 
and then process the stored image with no limit to the number of cycles that can be perf omned. Furthemnore, the logical 
component in the global analogic program unit (GAPU) of the CNN universal machine allows the device to adapt to 
the context, as well as the content, of the visual scene. 
25 [001 5] The CNN bionic eye and other topographic sensors would therefore be invaluable in a variety of applications, 
of which a few are: 

1 ) Serving as the "eye" for complex robotic vision systems capable of adapting to any given visual scene regardless 
of brightness, texture, contrast, frequency of light or speed of movement; 
30 2) Fomning the front end of visual prosthetic devices designed to interface at virtually any level of the nervous 

system from the optic nerve to the visual cortex and beyond; 

3) Satisfying a broad range of neuromorphic modeling objectives because the programming is so flexible and 

intuitive; 

4) Serving as the front end of a "smart" video or still camera capable of dynamically adjusting contrast, color 
35 composition and movement sensitivity of the scene even before the image reaches either the photodetectors or 

the film; and 

5) Serving as the lightweight, portable sensor in a fomi of bionic eyeglasses. 

[0016] A key point of the implementation is that all the sensors are placed on, or in close proximity to, the chip. In 
40 addition to the photosensors, pressure, temperature, and chemical microsensors could also be placed on the chip, as 
well as others. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0017] Figure 1 is a two-dimensional simplified block schematic diagram of a single-layer cellular neural network of 
the prior art showing a CNN having the interconnections between cells being only to the nearest neighbors. 
[0018] Figure 2 is a block schematic representation of the interconnection between the layers of a multi-layer CNN 
of the prior art. 

[0019] Figure 3a is a graphical illustration of a layer of a universal CNN machine of the prior art. 
50 [0020] Figure 3b is an expanded block diagram of a single cell of the prior art CNN machine of Figure 3a. 
[0021] Figure 3c is an expanded block diagram of the GAPU of the prior art CNN of Figure 3a. 
[0022] Figures 4a and 4b together present a schematic representation of an equivalent circuit of a single enhanced 
prior art CNN cell as in Figure 3b. 

[0023] Figure 5 is an expanded block diagram representation of GAPU of Figure 3c. 
55 [0024] Figure 6 is a simplified schematic representation of a representative eye and the layers of the retina. 

[0025] Figure 7a is a functional schematic representation of the layers of the retina of a representative eye and the 
processing of information by the various layers. 

[0026] Figure 7b is a simplified functional schematic representation of the layers of the CNN universal machine 
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programmed to simulate the processing of information by the various layers of the retina as shown in Figure 7a. 
[0027] Figure 7c is a simplified functional block diagram of the operation of the CNN universal machine of Figure 7b 
for simulation of the processing of information by the various layers of the retina of Figure 7a. 
[0028] Figure 8a is a schematic representation of the inclusion of a variable capacitance within each CNN cell. 
[0029] Figure 8b is a graphical representation of the capacitance value with respect to time of a programmabiy 
variable capacitor. 

[0030] Figure 8c is a schematic representation of the inclusion of a time-delay circuit, such as a variable capacitance 
In an R-C circuit, in the signal transmission path between two CNN layers. 

[0031] Figures 9a-9h are an original gray scale image is mapped into several other images by a CNN universal 
machine from the original gray scale image of Figure 9a using different analogic programming templates. 
[0032] Figure 1 0 is a diagrammatical representation of an analogic algorithm for combining various modalities in the 
same CNN universal machine. 

[0033] Figure 11 a is a schematic representation of single point measurement of a response of a CNN layer 
[0034] Figure 11b is a schematic representation of multiple point measurement of a response of a CNN layer. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

Summary of the Structure and Functions of a Representative Eye 

[0035] It is well known that biological retinas consist of sheets, or layers, of neurons arranged in very large arrays. 
Each sheet consists of between 1 0^ to 1 0^ neurons with the number of neurons per sheet dramatically decreasing as 
one's attention moves from the outer most sheet to the inner most sheet. Therefore it can be said that the grain of the 
retina changes dramatically as the inner most sheet is approached. 

[0036] Referring first to Figure 6, there is shown a simplified schematic representation of a representative eye and 
the layers of the retina. At the top of Figure 6 is the back wall, or sclera 1 34, of the eye, and at the bottom of the figure 
Is the lens 1 32 on the front of the eye. The retina of the eye, which is physically located near the sclera 134, is made- 
up of a series of layers which face away from lens 132. Thus, the incoming light 130 enters the eye through lens 132, 
passes completely through the retina, and then is in essence reflected off the sclera 134 before it Is processed by the 
retina layers. 

[0037] In this representation there are five layers in the retina, and each layer processes the light Information serially 
starting with the layer of photorecptors 136 which is made-up of the rods and cones. This first layer is the only layer 
that actually Interacts directly with the incoming light. Each of the remaining layers receives neurological impulses from 
the preceding layer, or layers, and It is those neurological impulses from the preceding layer that each subsequent 
layer operates on. Also, the deeper the layer of interest, the fewer cells that that layer contains, thus it is said that the 
"grain" of each layer increases the deeper the layer is. Stated another way, each cell in each subsequent layer of the 
retina receives its input signal from numerous cells (perhaps from 2 or 3 to 100, or more) in the previous layer of the 
retina. 

[0038] Referring again to Figure 6, following the layer of photorecptors 136, the layers in order are the horizontal 
cells 138, the bipolar cells 140, the amacrine cells 142 and the ganglion cells 144, with each of these layers performing 
a different function and modifying the neurological signal 146 from the previous layer to form the output signal 148 that 
Is presented to the brain for further processing. Thus it can be seen that the light and light Infomiation passes through 
the retina three times as signals 130, 146 and 148. 

[0039] Physiological recordings from intact retinas and retinal slices have yielded the functional parameters that 
underlie retinal function. These functional parameters are summarized in Figure 7a as 1 7 arrows, each representing 
a functional parameter of the eye. Figure 7a also identifies the five layers illustrated in Figure 6, and thus. Figure 7a 
schematically illustrates the function performed by each layer of the retina and Its immediate Interaction with another 
layer or layers. 

[0040] In Figure 7a the vertical arrows represent time constants and the horizontal arrows represent space constants 
which only occur in a single layer, not between layers; and the curved arrows represent transfer functions between 
layers. These 17 parameters specify overall retinal function and are well known in the literature. 
[0041] Modern retinal research is now beginning to uncover neuromodulatory modifications to basic retinal function, 
whereby the original functional parameters specified by the arrows In Figure 7a are being modified as more is being 
learned about the micro-operations of the various layers of the eye. These modifications allow a given retina to perform 
in many different modes depending upon intrinsic or e)ctrinslc conditions. For example, intrinsic conditions, are deter- 
mined by the biological clock, and the animal's state of awareness; external conditions are determined by the context 
of the visual scene, i.e. the brightness or contrast within the scene. 

[0042] In general terms, the 17 functions performed by the various layers of the eye are summarized below. First, 
the photoreceptors (136) transduce light and they become active neurally. 
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[0043] Next, the horizontal cells 138 average the activity at each point from input signals received from many-many 
rods and cones in the photoreceptor layer 136. Since each cell in layer 138 receives an input from many cells in layer 
1 36, that averaged image from layer 1 36 per cell in layer 1 38 is going to be very blurred because many points converge 
on each one. If the original image is dynamic, the result here is also blurred in time by the averaging action. 
5 [0044] The difference is then taken between the signals from the first and second functions. Thus, since the difference 
is being taken between a blurred image and a sharp image, the result is going to identify the edges of the original 
image of the incoming light 1 30. This difference taking, thus, enhances the edges and begins to bring out some of the 
details of the incoming image being received by tens 130. 

[0045] Then the difference signal is emphasized which In essence turns up the gain at the edges of the original 
10 image. At this point (bipolar cells 140) there are two kinds of cells: one type looks at increases in lightness; and the 
other type looks at increases in darkness. Thinking of a boundary between light and dark, those two events are always 
going to be neighboring each other with the actual edge being the zero-crossing between them and not otherwise 
defined. This can be thought of as the second spacial derivative of the original image. 

[0046] Up until this point, the retina has substantially been emphasizing spacial differences in the original image. At 
'5 this point on the retina changes in time have been identified. As a result of the next operation, the only portion of the 
original image that produces a resulting signal here is that portion of the original image that is moving. 
[0047] Next, there Is a kind of averaging in time that takes place with the motion detected in the previous step ap- 
pearing to be smeared out severely. 

[0048] Then finally, the ganglion layer 1 44 gathers this activity with the final output consisting of the changes in lime 
20 that have been identified the instant that they occur with the signal then being shut-off. 

Summary of CNN Universal Machine and Supercomputer Architecture 

[0049] Referring to Figure 1 , there is shown a 4x4 two-dimensional cellular neural network of the prior art by way of 
25 illustration. The cells C(i j) are shown here in a square array of four rows and four columns with "i" representing the 
row number and "j" representing the column number. In Figure 1 , there are sixteen cells each shown schematically as 
being interconnected only to its nearest neighbors, i.e., the first layer of neighbor cells. For example, cell 12 {C(2,2)} 
is shown interconnected with cells 30-44 by links 14-28, respectively. Each of links 14-28 shown here are bidirectional 
between the two cells to which that link is connected. In this example, there is only one layer of neighbor interconnects 
30 between the cells with cells 30-44 substantially forming a circle around cell 1 2. If there were a second layer of neighbor 
interconnects, cell 12 would similarly also be interconnected with cells 46-58. This layer of neighbor cells forms only 
a partial circle due to the limited size of the CNN in this example. 

[0050] Referring to Figure 2, there is shown a typical schematic representation of a multi-layer CNN of the prior art 
having layers 60-64. The input signal is shown being applied directly to layer 60 and feed forward path 68, and via 
35 delays 74 and 76 to layers 62 and 64, respectively. In addition, the output signal from layer 60 is applied to layer 62 
via delay 70. and the output signal from layer 62 is applied to layer 64 via delay 72. Finally, there is a feedback path 
66 for feeding back signals from layer 64 to layer 62, and from layer 62 to layer 60. In this implementation, the value 
of delay, 6, for each of delays 70-76 is substantially equal. 

[0051 1 The CNN universal machine of the present invention provides at least three new capabilities; the analog stored 
"^o program and the time-multiplex operation of layers, the wireless detection, and flexible representation of various PDEs 
using a chaos circuit. 

[0052] First, the CNN can be defined as an analog cellular nonlinear dynamic processor anray characterized by the 
following features: 

45 (i) analog processors that process continuous signals, that are continuous-time or discrete time signals (i.e.. the 

Input, output, and state variables are C*^ functions at least for finite Intervals), the processors are basically Identical 
(a single term of them may vary regularly); 

(ii) the processors are placed on a 3-dimensional geometric cellular grid (several 2-dimensional layers); 

(iii) the interactions between the processors are mainly local, I.e., each processor is interacting with others within 
^o a finite neighborhood (nearest neighbor is just a special case), and mainly translation invariant defined by the 

cloning template (the dynamics of a given interaction may be a design parameter); 

(iv) the mode of operation may be transient, equilibrium, periodic, chaotic, or combined with logic (without A-D 
conversion). 

55 [0053] The original CNN array 1 0 (see Figure 1 ), with some elementary logic additions, is used as a nucleus for the 
CNN Supercomputer with the following additional features and circuit elements being responsible for the universal 
capabilities of the CNN Supercomputer. Analog two or three dimensional array-signals which assume continuous values 
In continuous or discrete-time over an interval are used. 
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[0054] Referring to Figure 3a, there is shown a partial block diagram of the CNN universal chip 1 00. This chip archi- 
tecture is divided into two distinct regions as shown in Figure 3a. In the upper portion of Figure 3a there is what appears 
to be the CNN stnjcture shown in Figure 1 , and in the lower portion there is a block representation of the global analogical 
programming unit (GAPU) 102 which Is discussed further below. 

5 [0055] As is well known in the CNN art, the grid of individual cells 110 as shown in Figure 3a as being rectangular, 
could be any two-dimensional configuration (triangular, hexagonal, polygonal, etc.). This is more application dependent 
than theory dependent. Additionally, different grid sizes and shapes can be used for different layers in a multi-layer CNN . 
[0056] A connection radius of r=1 Is shown In the upper portion of Figure 3a for simplicity. It should be understood 
that, without loss of generality, larger radius values can be implemented In the same way to Increase the connection 

10 density as necessary. 

[0057] In fact, the universal chip architecture illustrated in Figure 3a represents the functional architecture of the 
CNN universal machine In two senses: (1) several CNN universal chips 100 (e.g. layers 60-64 In Figure 2) stacked 
upon each other results In a three dimensional universal CNN machine, and (2) several two-dimensional "layers" (as 
in Figure 1) of a three-dimensional CNN machine can be Implemented on the CNN universal chip architecture with 

15 time multiplexing using local analog and logic memory registers. To further broaden the cell capacity of a physical CNN 
universal chip, several chips can be placed next to each other on a planar substrate with only the border cells on one 
chip connected to border cells on an adjacent chip as if all of the cells had been on the same larger chip. If the feedback/ 
feedfon/vard signals of the CNN are to be connected to more than Just the nearest circle of neighbor cells then the 
border cells of each of the adjacent cells would also have to be connected to the appropriate additional cells on each 

20 chip as if those cells were each on the same chip. 

[0058] An expanded view of any and every CNN cell 1 1 0 of Figure 3a is shown in more detail in Figure 3b. In addition 
to the nucleus of the single cell which Is equivalent to the basic CNN cell, there are several additional analog and logical 
circuit elements associated with each cell. It should be noted here that "logical" and "digital" are not to be equated with 
each other. A digital circuit includes various gates and registers, whereas a logical circuit is an analog circuit that 

25 processes logical values without doing so with conventional digital building blocks. 

[0059] The additional circuit elements in each cell of the CNN universal machine are a local analog memory unit 
(LAM) 112, a local logic memory register (LLM) 114, a local logic unit (LLU) 116, a local communications and control 
unit (LCCU) 118, and a local analog output unit (LAOU) 120. In operation the LAM 112 cell values are combined by 
the LAOU 120 while the logic register values are combined by the LLU 116. The LCCU 118 provides the control of the 

30 cell configuration and the analog and logic instruction communication. The operation of the enhanced universal CNN 
cell and each of these elements is discussed in greater detail with respect to Figures 4a and 4b below. 
[0060] The global analogical programming unit (GAPU) 102 controls the entire cell and is shown in more detail in 
Figure 3c. Here it can be seen that GAPU 102 has three constituent parts, an analog program register (APR) 104, a 
logical program register (LPR) 106, and a global analogical control unit (GACU) 108. Thus it can be seen that GAPU 

35 102 is a "dual" unit with both analog and logical components, but without A-D converters. It is used to control the 
identical functions of all cells and the sequences of these functions, namely: 

• the analog template elements; 

• the function of LLU 116; 

40 • the cell configurations (including the selection within LAOU 1 20); 

• the start of the different units. 

The operation of GAPU 102 and each of the elements that it contains will be discussed in relation to Figure 5 below. 
[0061] If the implementation of the overall CNN universal machine is done on a single chip, then several of GAPUs 
45 102 can be used in an array partitioning GAPU 102 between the chips working together with a central GAPU 102. In 
such an Implementation APR 104 can also be divided to form an analog cache register. 

[0062] Figures 4a and 4b together present a schematic diagram of an equivalent circuit representation of a typical 
individual cell 1 1 0 (see Figure 3b). In this diagram the suffixes u, x and y denote the input, state and output, respectively, 
of the cell. Additionally, node voltage v^y is called the state of the cell, node voltage v^y is called the input of the cell, 

50 the node voltage v^y is called the output of the analog portion of the cell, and is the logical output of the cell. 

[0063] Templates A and B define the feedback and feedforward, respectively, between neighbor cells and are pro- 
grammable via GAPU 102. They each contain variable gain amplifiers with the gain values being received externally. 
At the input, sequentially, various values of node voltage v^y, the input signals, are stored as analog signals locally in 
LAM1 112^. Similarly, the initial state, v^y(O), is also stored locally as an analog signal in 1-AM2 112^, and subsequent 

55 state values are store sequentially as analog signals in IJVM3 1 12^. Finally, several consecutive output values, Vyjj. are 
stored as analog signals in LAM4 112^. Once at least one result is present in LAM4 112*, it can be used as an input 
or initial state to LAOU 120, meanwhile different B and A templates can be programmed to calculate iterative instruc- 
tions. In very high speed applications LAM4 11 Z* could be an analog delay line. Now, after several consecutive results 
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have been sorted in LAM4 112^, they are combined by LAOU 120 to calculate an output value, v'yy. For example, 
LAOU 120 could be a diode or a small amplifier which adds the analog values from LAM4 112*. One implementation 
of LAMs 112 could be capacitors on which the analog voltage can be stored. 

[0064] In the local analog memory (LAM) 112^^, in addition to the local input, state, and output memory elements, 
5 ■ some additional output memory cells are used in LAM4 112*. These additional output memory cells provide for the 
application of a local analog output unit (LAOU 120) where the analog memory values are combined by simple analog 
circuits (using programmable switches one particular operator can be selected). These elements can also be used (in 
an analogical algorithm) for the implementation of branching into several parallel branches (time multiplexing with 
analog signal values). 

10 [0065] From the analog portion of the CNN of the present invention that is shown in Figure 4a the signal v*y|j, in 
Figure 4b, is applied to a simple bipolar/unipolar shift unit 122 transfonris the analog output to logic values with con- 
secutive values being stored in a local logic memory register (LLM) 114 which may be implemented, for example, by 
a multi-stage analog shift register where the converted logic signals are stored in analog fonn (just yes or no). Having 
several consecutive logic values stored in LLM 114 allows for the implementations of branching in the analogical al- 

15 gorithms (at logic signal arrays). 

[0066] The results stored in LLM 1 1 4 are then cornbined in LLU 116. LLU 1 1 6 is programmable to combine the logic 
values stored in LLM 114 into a single output. Further, LLU 116 is programmable from GAPU 102 and may be imple- 
mented, for example, e.g., as a memory or a PLA. Thus the logical output signal, Yy, provides a "yes" or "no" response 
to whether the feature that the CNN was programmed to detect by templates A and B was observed. For example, 

20 vertical edges and corners and where they occurred might be detected. The analog-logical (analogical) processing, 
. as illustrated in Figures 4a and 4b, is serial with the analog processing performed first followed by the logical processing. 

[0067] Throughout the analogical processing discussed above with respect to Figures 4a and 4b, LCCU 118 controls 
the timing of the various steps in the processing by means of a series of switches that are controlled by GAPU 102. 
These switches interface with the various elements of Figure 4a via the broken line segments shown in that figure thus 

25 controlling the operation of LAMs 112. The switches of LCCU 118 can be implemented several ways including, for 
example, MOS transistors or SCRs, with the switches being locally the same in each cell. 

[0068] These configurations are the same for all cells. If there Is a global control unit which sends the switch config- 
urations to each cell, the same binary switch configuration code is sent to each cell. Then when the next instruction 
comes a different switch configuration code will be sent. 

30 [0069] In the previous design of the original CNN it was necessary to have multiple layers, each working in parallel 
and each looking for the occurrence of a different feature. This approach is still possible with the universal CNN, 
however, since the universal CNN Is programmable it is possible to use the same hardware to search for different 
features in a serial mode which greatly reduces the Interface hardware necessary for implementation of the universal 
CNN supercomputer where all of the necessary functions can be performed serially within the available time. Since 

35 the CNN supercomputer operates in very fast in real time, and if there are five things that need to be searched for, the 
CNN supercomputer is cycled through five times. The key here is that it Is so fast that it appears to be nearly instan- 
taneous. When there are so many stepsthat have to be performed within a fixed amount of time, It may still be necessary 
to use multiple layers of the universal CNN supercomputer that function in parallel. 

[0070] Since the CNN supercomputer employees multiplexing and the results are locally stored in each layer, the 
40 results from several layers that have been processed simultaneously can be combined with no communication cost 
since everything is locally stored. This is key. Also analog signals are used so that there is no need for A-D conversions 
In each cell; the actual signals are stored. The local connectivity here provides a high speed supercomputer that is 
very small and uses very little power. 

[0071 ] Figure 5 is a block diagram representation of GAPU 1 02 which as stated in relation to Figure 3c has three parts: 

45 

the global analog program register (APR) 104 for storing the analog programming instructions and different tem- 
plates for each cell which are the same for each cell (e.g. by using capacitor arrays); 

the global logic program register (LPR) 106 for storing the different logic programming instruction and codes of 
LLU 116 (same for all cells in the layer); and 
so the global analogical control unit (GACU) 1 08 which controls the overall timing for the CNN layer and when each 

instruction from APR 1 04 and LPR 1 06 are entered into l_AM 112 and LLU 116, respectively of each cell (the control 
sequences stored here represent the analogical machine codes of the analogical CNN algorithm) with the use of 
two global clocks, one determined by the analog dynamics and the other determined by the logk: units. 

55 [0072] To overcome the on-chip size limitations of these 3 subunits, especially the size limitations of APR 104, the 
division of the on-chip and off -chip analog program storage can be arranged to introduce analog-cache storage (ACS) 
as well to supplement the on-chip APR 104 storage. 

[0073] The analogical CNN algorithms can be described by an analogical CNN language (or an equivalent block 
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diagram). This language is translated by a computer to the analogical CNN machine code. 

[0074] In most applications the radius of interconnects with each CNN will be equal to 1 , thus there will be no more 
than 1 9 individual instructions to be stored in the registers of APR 1 04, thus 1 9 numbers will be enough to fully address 
the processor. Each of the APR 104 registers of Figure 5 represent different analog instructions. These instructions 
consist of all of the possible values for each of the A and B templates, as well as the bias current, I. The bias current 
can also be thought of as a threshold. Each contains an instruction to program the individual templates: "A" with 9 
numbers (3x3 matrix with values for feedback to the connected cells of the CNN), "B" with 9 numbers (3x3 matrix with 
values for feedforward to the connected cells of the CNN), and "I" with 1 number for the bias current of the single cell 
Itself. Similarly, if r=2 then 51 numbers will be needed since the "A" and "B" templates each will be 5x5, if r=3 then 99 
numbers since the "A" and "B" templates each will be 7x7, etc. 

[0075] Similarly the size and number of each register in LPR 106 is dependent upon the number of logic values to 
be combined and the number of different logic instructions. 

[0076] The selection of the appropriate analog templates, i.e. analog instructions^ and the logical instructions is made 
by GACU 108. Suppose the next instruction is a corner detecting template. GAPU 108 then loads to the local program- 
ming values of the "A", "8" and "I" templates. Like in digital microprocessors the contour here is organized in the 
sequence of the data and instructions. GAPU 108 is a sequential logical store of which instructions to select and in 
what order from APR 1 04 and LPR 1 06 to detect the desired pattern, etc. 

[0077] Considering the standard fully connected analog neural networks, it is obvious that the latter condition is 
unimplementable. Partly because the storage space of a single program "instruction" is prohibitively large, partly be- 
cause the reprogramablllty takes more time than the execution time itself. This Is a point where the local connectivity 
comes into the picture. Namely, if a CNN cloning template is considered an analog instruction, then there are only one 
or two dozen analog values for an instruction to be stored even if there are thousands of processors. Conversely, in 
the case of a fully connected analog synchronous neural network (e.g., the Intel 801 70), there are about 1 0,000 analog 
values for 64 processors and their reprogramablllty takes considerable time. In addition, the possibility of using different 
artificial dynamic systems, or circuits, as cells (e.g. a chaos circuit), makes the array used in the present invention 
extremely flexible. 

CNN Implementation of the Retina - The CNN Bionic Eye 

[0078] In order to implement retinal function in a CNN universal machine, the time and space constants, as well as 
the transfer functions, and modes of adaptive behavior for retinal Image processing must be identified from the data 
gather through years of biological research that Is already available In the published literature and are summarized in 
the 17 parameters displayed diagrammatlcally In Figure 7a. Given these physiological and physical parameters, the 
CNN templates, etc., can be derived and then programmed into the CNN universal machine in the following ways. The 
key is how to translate them Into the CNN universal chip. The CNN literature contains several papers on the "learning" 
aspects of CNN to relatively automatically generate the necessary "A" and "B" templates to perform desired functions. 
Additionally, also In the literature, some of the published biological models contain matrices to explain various opera- 
tions of sensory organs which can be used will little variation to program certain functions in the CNN universal machine. 
[0079] In addition to the identification of the basic CNN structure, the following design steps are essential: 

1 . Time constants and delays are defined for the CNN universal machine by controllable capacitors, small delay 
circuits, and by analogic programs for modifying both global and local time constants; 

2. Spatio-temporal properties are Identified by specific CNN template design methods for embedding the space- 
time parameters derived from retinal function in the CNN universal machine (there are two template design strat- 
egies based on single and multiple point measurements - In the single point case the excitation place corresponds 
to the template element location, while in the multiple point case a learning algorithm Is used); 

3. Programmable compression and extension techniques are used for matching the "grain" of the CNN universal 
machine elements in a particular operation simulation to the grain of the neurons of the retina or other sensor at 
the various retinal layers; and 

4. The programming of the global analogic program unit (GAPU 1 02 in Figure 3a) for including the adaptive prop- 
erties of retinal, or other sensory, function. 

[0080] Combining several CNN simulated retinas for more complex tasks by using the analogic software of the CNN 
universal chips are the next level of the design. In addition, several different topographic sensory modalities can also 
be combined. 

[0081] The CNN universal machine Is powerful and flexible enough to be used to match the function of alt known 

vertebrate retinas, and can easily perform other functions that go far beyond biological possibilities. 

[0082] For example the base functions of contrast detection, color coding, movement detection, directional selectivity 
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can be easily perfomned by the CNN universal machine on a single chip just by programming using analogic software. 
Most Important, these functional properties can be modified according to the brightness, texture, contrast and rate of 
movement in the visual scene, and this modification can take place either locally or globally. In this way the CNN 
universal machine can be programmed to operate at the most efficient level as determined by both the instantaneous 

5 ' visual scene and the long range visual goal. 

[0083] An example of the capabilities of the CNN universal machine which lends itself to use as a bionic eye is 
illustrated in the composite of 11 different projections of the same input image (upper left) in Figure 9 using 11 different 
analogic instructions, with any or all of these projections being generated and stored In memory within microseconds. 
Just as easily and quickly, any one of these images could have been used as the input image for the next operation, 

10 as occurs in the retina of an eye. 

Transferring the Architecture from the Published Physiological Databank to 
CNN Universal Machine. 

15 

[0084] The CNN bionic eye is programmed using the known physiological parameters arid models 'of retinal function. 
Physiological recordings from intact retinas and retinal slices have yielded the functional parameters that underlie 
retinal function. Figure 7a, as discussed above, shows a schematic diagram of the retina, along with 17 arrows, each 
representing a functional parameter. This structure forms the basis of the implementation of a CNN bionic eye. The 
20 vertical arrows represent time constants, the horizontal arrows represent spacial constants, and the curved arrows 
represent transfer functions. This set of parameters specifies overall retinal function whrch has been illustrated in the 
literature by implementation on a digital image processor. 

[0085] Modern retinal research is now beginning to uncover neuromodulatory modifications to basic retinal function, 
whereby the original functional parameters specified by the arrows in Figure 7a can be further modified to further 
25 enhance effective bionic implementation of retinal function. These modifications allow a given retina to perform in many 
different modes depending upon intrinsic or extrinsic conditions. For example, intrinsic conditions are determined by 
the biological clock, and the animal's state of awareness; extrinsic conditions are determined by the context of the 
visual scene, i.e. the brightness or contrast within the scene. 

[0086] The CNN bionic eye can also adjust to both intrinsic and extrinsic properties of the visual scene. By basing 
30 the programming of the CNN bionic eye on physiological parameters, the bionic eye will be endowed with the Instan- 
taneous perfomnance characteristics that follow biological retinal function. These characteristics are further subject to 
modulation as a function of extrinsic properties via the global analogic program unit (GAPU 102 in Figure 3a), similar 
to the way biological retinas modulate perfomiance characteristics. 

[0087] Figure 9a shows an original gray scale image that was processed by the CNN universal machine with different 
j5 «^H^ a|. templates as shown in Table I to generate Figures 9b-h. Briefly: 

Figure 9b is derived from Figure 9a using a half-toning set of templates; 
Figure 9c is derived from Figure 9a using an averaging set of templates; 
Figure 9d is derived from Figure 9c using an edge set of templates; 
40 Figure 9e is derived from Figure 9a using an on-center, off-surround set of templates; 

Figure 9f is derived from Figure 9a using a diaggrey 135 set of templates; 
Figure 9g is derived from Figure 9a using a diaggrey 45 set of templates; and 
Figure 9h is derived from Figures 9f and 9g using a logic set of templates. 

45 [0088] In summary, the CNN layer and interconnection architecture Is started by a multilayer CNN with as many 
layers as physiological ones. The template stnjcture is derived to mimic directly the receptive field organizations of the 
retina. In the case of between-layer feedback, all layers which are involved should be considered together. Feedforward 
layers are considered sequentially, by time multiplexing the consecutive layers in a programmable way. If more than 
one layer is connected by feedback, complex CNN cells are used in the physical CNN implementation. 

50 

Implementing the Physiologically Faithful Space-Time Parameters and Interaction Characteristics 

[0089] In order to implement retinal parameters in a CNN universal machine, the sites of interconnections, the inter- 
actions (transfer functions), and nonlinear dynamics between all elements must be specified. Unlike phenomenological 
55 models (which can also be modelled by CNN universal chips), here the physiologically faithful models are represented 
on a CNN universal chip. 

1) Basic time constants and delays of retinal function are set by the controllable capacitive concept or by analogic 
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programs, 

2) The space constants and spatio-temporal interactions of retinal function or other sensory modality are set by 
programmable "templates" that define the interconnections between layers and cells in the CNN. (Two design 
methods are specified) . 

5 3) There is a form of layer compression; in the retina, over 1 million photoreceptors converge to less than 10,000 

output cells. Therefore the CNN universal machine must be capable of layer compression. The compression of 
layers is accomplished by deleting or time-multiplexing rows/columns in the CNN layer grid by modifying the an- 
alogic program. 

4) The adaptive properties of retinal (or other sensory) function, whereby retinal parameters are modified to ac- 
10 commodate different scenarios or processing goals, are implemented by the analog capacitive and logic bi-stable 

storage in the global analogic program unit (GAPU 102 in Figure 3a) of the CNN universal machine. These adap- 
tations can be programmed based on either content or context. 



15 



25 



[0090] Even the switch between different • retina models can be perfomned adaptively. 
Design Innovations in Detail 



[0091] The CNN universal machine has the computational power to perform highly complex processing of visual 
images with great speed, consuming little power, while only occupying the space of a computer chip. The programming 
20 of the operation of a CNN has great flexibility. It can be externally programmed to perfonn an enonrjous variety of 
image processing functions, feed the results of its processing back to Itself so that it can reprogram itself to adapt to 
different qualities of the image or scene, and its operation can be reset as a function of the requirements of its host. 
[0092] In order to implement the CNN universal machine as a CNN bionic eye, some key design additions, listed 
above, must be made to the design. These are as follows. 



Controllable R-C Time Constants and Time Delays 



[0093] Biological retinas act primarily to abstract information about the dynamics of the visual scene. Movement, 
direction of movement, onset and termination of local and global events, expanding and contracting images associated 

50 with, for example, approaching targets, are allcommon elements of visual function. In orderto adequately and efficiently 
process the dynamics of the visual scene with an electronic model, a time delay Is sometimes necessary between 
various functions, or steps in the process. When the necessary time delay Is relatively short at a particular equivalent 
CNN layer, or a complex CNN layerthat simulates the function of more than one layerof a retina, a controllable capacitor 
in an R-C circuit or an FET can be included on the CNN bionic eye chip to provide the ability to provide a programmed 

35 variable time delay. For global and longer time delays than are feasible with capacitors because of their necessary 
size, the time delay can be programmed into the operational routine and stored in the local analog memory of the 
affected CNN cells (LAM 112 of Figure 3b). Thus, by using one of the two Implementations of delays discussed above, 
It Is possible to modify global time constants and/or time delays of different CNN layers, or to modify the time constants 
and/or time delays at local positions. All of these time constants and time delays being programmable, therefore, they 

40 too can contribute to the adaptatlonal properties of the CNN bionic eye. 

[0094] Identical controllable capacitors, C^, can be placed on the CNN chip to control the speed of the basic dynamics 
(the time constant) of the CNN array. Figure 8a Illustrates the placement of the capacitor, C^^, in the front end of the 
individual CNN circuit of Figure 4a. The capacitors, C^, may vary In a preprogrammed way (part of an analogic program 
of the CNN) or continuously depending upon the local output voltages of CNN cells on different layers (see Figure 8b). 

45 [0095] Controllable capacitors can also be used to change the time delay of a template element. Rgure 8c shows a 
circuit implementation In the transmission path between CNN layers with a variable capacitor or delay-line inserted in 
that path. Hence, different delay values between the CNN layers and in the different positions of either "A" or "B" 
template can be Implemented. Delay templates can also be useful in many motion- related applications. 
[0096] Thus, If the capacitive variation can be controlled by a change in voltage level the time constant can easily 

50 be Implemented on an IC (CNN universal machine). This provides the option of placing a variable capacitor, or other 
time delay mechan ism, on the Si chip itself and simply varying the delay time by programmably varying a control voltage 
level. 

[0097] For larger delay values (compared to the "R-C time constant" discussed above) the local analog memory cells 
(l_AM 1 1 2 of Figure 3b) can be used to store a signal value that is applied to the appropriate CNN layer by the analogic 
55 program at the appropriate time. In this case a second "A" or "B" template circuit (bank of OTAs -Operational Transcon- 
ductance Amplifiers) can be Implemented, switching them in under the control of programmable switches, as part of 
the switch configuration of the CNN universal chip. If more than one layer is to be effected In the same way or at the 
same time, then those CNN cells can be placed on a single physical layer as complex CNN cells. 
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Identifying the CNN Templates Based on Measurements in the Living Retina. 

[0098] The key to making the CNN pertomi like a biological retina is to create a correspondence between the CNN 
templates and measured interactions in living retinas. Rather than defining the properties of Individual cells, information 
about retinal networks has been extracted from the literature. Spatial interactions are defined in tenns of a "space 
constant" that describes the lateral spread or diffusion of activity. This diffusion is then incorporated into the design of 
the cloning template of the CNN. Similar procedures are followed for implementing time constants and transfer functions 
for the bionic eye. The 1 7 parameters that constitute the design properties of the retina are illustrated in Figure 7a above. 
[0099] Figure 7b is a simplified functional schematic representation of the layers of the CNN universal machine 
programmed to simulate the processing of information by the various layers of the retina as shown in Figure 7a. In ■ 
Figure 7b Layers 2-5 (1 38-1 44) of Figure 7a are illustrated as individual CNN Layers 1 50-1 56 and identified on the left 
with the same temiinology used in Figure 7a. Layer 1 (136) of Figure 7a Is Illustrated here as the photoreceptors that 
gather the Image and apply it to the layer of horizontal cells 138 in the retina, represented here as CNN cells 150 in 
Layer 2. Also, as discussed in relation to the operation of the retina where there are fewer and fewer operational cells 
in each layer of the retina as the signals progress from the first layer to the fifth layer, for illustrative purposes that 
reduction in the number of cell layer by layer is shown by there being only half as many CNN cell 150 being shown in 
each layer compared to the number of cells in the previous layer. The factor of 14 was selected only for representational 
purposes when in actuality the variation from layer to layer varies, and it varies from animal to animal. 
[0100] Figure 7c then shows the corresponding functional block diagram of the operation of the simplified CNN 
universal machine of Figure 7b with each of blocks 160-166 programmed to function according to the parameters 
related to the corresponding layer of the retina of Figure 7a. 

[0101] Single point measurements In all physiological layers can be used as responses to the on-center and off- 
center light stimuli. A secondary algorithm can be derived to adjust the different template elements for on-center and 
corresponding off-center images. Normally, these adjustments are performed in a heuristic way, taking into account 
the physiological information as well as excitation-inhibition, etc. Using a single point measurement and a moving light 
spot the different template element dynamics or aggregated interactions between neighboring cells (using a larger light 
spot) can be measured or determined from the literature on the retina, or other sensory organs. 
[0102] By combining these techniques with multiple grid point measurements in living retinas while applying given 
single or multiple input light illumination pattems, conventional CNN template design methods can also be used. In the 
simplest case, the input-output patterns from these experiments can be fed to known template leaming algorithms, e. 
g. a genetic template learning algorithm. In this way the receptive field organization of retina or another sensory organ 
can be "reverse engineered". 

[01 03] Figures 1 1 a and lib illustrate these single point multiple point measurements on a CNN layer 1 86 containing 
an array of individual CNN celts 148. in Figure 11a the single point measurement technique is illustrated with the 
excitation point 1 88 (shown as "•") a being applied In the upper left corner cell (cell 1 1 of the upper left r=1 neighborhood 
of cells) and the measurement point (shown as "x") in the next to the lowest tier cells that is the second in from the 
right (cell 33 of the upper left r = 1 neighborhood of cells). The relative geometry of the excitation/measurement points 
are the same as that of the "A" and "B" templates. If either "A" or "B"* is all zeros, then the measurement is done directly 
at the a 11 or b^^ element in the r=1 neighborhood of cells. Assuming that r=1 and the "A" and "B" templates are 
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If both "A" and "B" at a given position (e.g. 23) are both non-zero, then it is necessary to make heuristic adjustments 
using CNN simulators to identify the combined effects, or several single point measurements are made and then the 

50 "learning" algorithms are used to calculate the "A" and "B" template values. 

[0104] In Figure lib the r = 1 neighborhood of cells is shown centered in CNN layer 186 with each of the cells in the 
neighborhood being measurement cells (denoted by the inclusion of an "x" in each cell) and only the cells surrounding 
the center cell being excitation cells (denoted by the inclusion of an In those cells). In a multiple point measurement, 
as illustrated here, the input/output pairs are measured and fed into the "learning" algorithm (e.g. a genetic template 

55 learning cell) where the "A" and "B" template values are calculated. 
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Programmable Layer Compression and Expansion. 

[0105] Biological retinas consist of sheets, or layers, of neurons, arranged in very large arrays existing of between 
1 million or 1 00,000 neurons. The number of element decreases dramatically as one moves from the outer to the Inner 
layers of the retina. Therefore the grain of the retina changes^ dynamically. In order to account for the changes in grain, 
a mechanism for programming layer compression or expansion, whereby the number of elements and the distance 
between these elements can be modified in the program, must be included In the CNN bionic eye. 
[0106] In the CNN universal chip, the functions of layer compression and expansion can be performed for grain of 
layers by using a physical CNN grid with the number of cells corresponding to the finest grain (the most CNN or retinal 
cells), with some of the rows and columns of the CNN chip programmed to switch off to simulate other layers of the 
retina with a coarser grain. 

[01 07] Also, the function of a single biological layer can be broken into sub-steps and modeled with more CNN layers 
(e.g., for the outer plexiform layer of the retina, 2 CNN layers can be used, one to model the interactions between the 
photoreceptors and the second for the horizontal cells) or the function of several biological layers can be lumped into 
a single CNN layer (e.g., the function of the photoreceptors, the outer plexiform layer, and the Inner plexifonn layers). 
[0108] The compression of the function of retinal layers is done in the CNN universal machine by deleting the time 
multiplexing In the multi-layer CNN analogic algorithm. Another technique is to use, complex cells (I.e. 2-3 layers phys- 
ically compressed by grouping the cells in Identical positions on different layers into a single complex cell, which Is 
then placed on a single physical layer on silicon). 

[0109] Further extensions of layers can be implemented by time multiplexing of the layer dynamics and assigning 
additional local memory units (LAM 112 or LLM 114 in Figure 3b). 

Mechanisms of Adaptation 

[0110] Biological retina function is not fixed, but can vary as the visual scene or the goals of the host are changed. 
For example, the "gain" of the retina is decreased when brightness of the visual scene increases. Gain is also decreased 
in local regions when the texture or contrast in those regions increases. These gain changes keep the retinal output 
within an acceptably narrow range of magnitudes, regardless of input conditions. Similarly, it is thought that biological 
retinal function can vary according to conditions of stress or excitement. In the CNN bionic eye, output from the eye 
Itself can be directed to the GAPU (1 02 of Figure 3c). which in tum can modify the behavior of the CNN bionic eye. 
[01 1 1 ] The adaptation may be controlled by intrinsic conditions, for example, when the difference between two con- 
secutive values at some given test cell is higher than a prescribed value the value of capacitor, C^, could be decreased, 
thus adapting the speed of the circuit to the variation of the actual scene. Another example is the adaptation to local 
illumination within a finite radius where an illumination sensing template can control the local threshold bias term, lij. 
[01 12] Extrinsic conditions or direct control of the analogic algorithms represented in the global analogic control unit 
GACU (108 of Figure 3c) can also be implemented. In this way, the context of a pattern may govern the adaptation. A 
prescribed sequence of templates and the logical combination of their results is a standard way of detecting complex 
events. For example, a template can be used for detecting horizontal lines, another for detecting red areas, and then 
the logical combination of these two templates will show only horizontal lines in red areas. 

[0113] A given sequence of a small number of templates can be implemented without the analog program register 
(APR 104 in Figure 3c) by locally switching between different transconductance values. Hence, the global wires con- 
nected to APR (104 in Figure 3c) can be deleted and the coded switch sequences stored in SCR (115 in Figure 5). 
Such complex cells can also be used in implementing color processing. 

Alternative Implementations 

[0114] Since the retina of the eye of each animal has only a few layers with most of the interaction between layers 
being between those layers that are physically adjacent to each other, the actual operation of the retina can be said 
to be substantially serial. That being the case, the CNN bionic eye modelling of the operation of the blologk^al retinal 
function could be perfonned with a single CNN chip. This can easily be accomplished since the CNN cell functions so 
much faster than the retina of a biological eye and since there are only a small number of variables that are necessary 
to program each function performed by each layer of a biological retina. Thus, the initial inputs ("A", "B" and "I" template 
values) for the CNN bionic eye are derived from at least one optical sensor that is viewing the desired scene. Then 
the results of the simulation of the first function of a biological retina can be stored in memory. From those stored values 
the analogic program can calculate the "A", "B" and "1" template values for the next retinal layer to be simulated. This 
process can then be repeated for each of the retinal functions with each reprogramming of the single CNN chip being 
a "virtual layer" and thus performing the same function as discussed above where multiple layers CNN chips were 
utilized. In summary then, by using a single CNN chip the analogic program calculates the values for one layer and 
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then locally stores those analog values, pixel by pixel, that result from the function performed at that "layer" is used to 
select the necessary templates for the next "layer", and then continue similarly "layer" by "layer". 
[0115] Because of the memories, L_AM (112 in Figure 3b) in the CNN cell, if there are photoreceptors (e.g. CCDs) 
on the IC chip, then the result (e.g. the illumination level of each pixel) can be stored. Assume that the output of that 

5 CNN "layer is of the simulated rods and cones, then the CNN chip is re-programmed to represent the function of the 
next layer of the retina, and so on and so forth. Because the actual length of time that It takes the eye to process the 
information through all of its layers is some much longer than the processing time of the CNN universal machine that 
is programmed as a bionic eye, the simulated processing can be done with a single CNN chip that is programmed in 
"virtual layers" to perform each of the necessary steps to fully model retinal function. This then requires time delays in 

io the CNN process, as discussed above, to fully simulate that performance of the eye. 

[0116] In essence what is happening in the CNN bionic eye that is implemented with a single CNN chip is that the 
processed signals at each step, or "virtual layer", are stored in memory, then from those stored signals new "A" and 
"B" templates to simulate the next function are derived in the computing unit under control of the analogic program, 
and the stored data from the previous "layer" is processed. The result of the next processing step is then stored in 

15 memory, and the parameters for the next process step are derived and that data is again processed, and so on for 
each function of the retina. 

[0117] If the biological function being simulated includes one or more steps, where data resulting from more than 
one previous layer is necessary for some computation, it would only be necessary to increase the size of the local 
memory so that multiple pixel values can be stored. Alternatively, if the situation is such, as it is with the use of the 
20 CNN universal machine to model the performance of the eye, that the biological processing time is much longer than 
the CNN processing time to perform the same function, it would be possible to store and use the pixel values from 
multiple layer functions to calculate the necessary "A" and "B" template values for a subsequent function and still be 
able to use only a single CNN chip. 

[0118] If the vast difference in time between the functional speed of the organ to be modelled and the CNN chip is 
25 not available, then multiple CNN chips could be interconnected to perform at least some of the functions either serially, 
or in parallel, or in both. Also, some portions of the program may be independent of the previous result in which case 
the program need not interact with the memory, and the program proceeds with the selection of the next templates to 
perfonm the next function, accordingly. 

30 Combining Several CNN Bionic Retinas and Modalities 

[0119] A given set of bionic retinas implemented as described above, can be combined in a prescribed or adaptive 
way. Since the retinal models of different species have some similar parts/layers, the templates common in more than 
one bionic. retina can be stored in the analog program register, APR (104 in Figure 3c), only once. Hence, the size of 
35 the APR (104 in Figure 3c) needs only to be equal to the number of different templates for all species, and other 
desirable functions. 

[0120] Additional local output memories {LAM 112 or LLM 114 in Figure 3b) can be used to store the results of 
different bionic retinas of different species. Thus, these different resulting images can be combined locally without 
reading out all the different output images. In this way, different capabilities of different species are combined. 

40 [0121] The combination of different CNN bionic retinas can be controlled by the GACU (108 of Figure 3c) in the same 
way the adaptation of a single CNN bionic retina has been described above. Thus, the analogic program stored in the 
GACU (1 08 of Figure 3c) can program the switches between the bionic retinas of different species within a microsecond. 
[0122] Other topographic sensory modalities can be represented in the same way as vision. Using microsensors 
attached to all CNN cells, pressure (or pressure difference), temperature, chemical parameters, etc., all the other 

45 sensory information, can be processed by a single CNN universal chip. These 2-dimensional sensor arrays can be 
designed in the same way as the CNN bionic eye, however, here the "images" and "maps" represent pressure-maps, 
temperature-maps, etc. 

[0123] The physical implementation of the multiple modalities can be on a single or multiple chips. In the first case, 
the different microsensors are to be integrated on each cell of the CNN universal chip. In the latter case, the particular 
50 ability of the CNN universal chip to interface directly with digital systems provides an excellent way to perfonm this 

combination. 

[0124] Figure 10 illustrates the combination of several different functions and modalities in the same CNN universal 
machine. In Figure 1 0 there are fourfunctions shown with an indication that additional ones could also be incorporated. 
Shown here is a bionic eye for daylight vision (174), a bionic eye for high speed horizontal motion detection (176), a 
55 bionic taste analyzer (1 78), and a bionic sensorto measure velocity and the acceleration rateof objects moving vertically 
upward (180), with each of these modalities under the control of GAPU 102. Also shown are a visual sensor 182 for 
providing input information to blocks 1 74, 1 76 and 1 80, and a liquid sampler 1 84 for providing input infomiation to block 
178. As discussed above, each of blocks 174-180 could each be implemented on one, or a number of CNN layers, or 
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virtual layers. It Is also possible that all of these functions, and many more could be implemented as virtual layers within 
the same CNN virtual machine given the vast relative speed difference between the CNN universal machine and bio- 
logical systems. 

Sample Applications 

[0125] In addition to the applications described at various locations above, the CNN bionic eye, with its multiple 
functionalities, is Ideally applicable to many medical diagnosis and screening tasks. Detecting slight defects, enhancing 
the contours of many medical imaging devices (especially in the less destructive ones, e.g. ultrasound devices), and 
extracting selections, are just a few. 

[0126] Remote sensing is an other area where mimicking different bird's eye views can be implemented. 
[0127] Combining visual and tactile information In robots exploits the local analog memory of the CNN bionic eye 
and tactile sensor. Combining visual and themrjographic images, as well as images at different illumination conditions 
(e.g. infrared) are other possibilities. 

[0128] The CNN bionic eye, as well as a CNN bionic fonn of other organs, could also be used to simulate normal 
and diseased functioning of those organs, and other complicated medical or physical conditions to permit research 
using the CNN bionic system to simulate potential corrections for the problem. For example, the CNN bionic eye can 
simulate complicated conditions such as spiral wave distortions in the eye which could lead to corrections for that type 
of problem without having to use living volunteers. A similar system might also be used to simulate problems such as 
that which exists with the Hubble telescope which could quickly lead to a "fix" which might be delivered to the telescope 
while in orbit without having to bring the telescope back to Earth, or having to build and launch another one. 
[0129] Night vision could be enhanced with the addition of the ability to see In the infra-red spectrum. Also the ability 
to measure the rate of movement that Is being observed since the human eye-brain can detect motion, but it can not 
measure it. With the CNN bionic eye with the proper templates, could actually measure rates of motion, as could angular 
velocity. Further, It might be useful to detect certain rates of motion but not others, or motion in a particular direction — 
the human eye does not make those kinds of distinctions, humans can detect just about everything in the visual spec- 
trum. Other things that might prove useful in some situations might also be the ability to see only certain limited light 
frequencies or to see only horizontal motion. 

[0130] What the human brain does is interpret what information it receives. In a bionic eye there could be many 
layers beyond those that simulate retinal function which perform interpretive functions far beyond what any animal can 
naturally do. 

[0131] The second part of the invention is that the CNN bionic eye makes it possible to combine the function of 
multiple eyes from different animals, or to combine selected functions from several animals, or even to combine func- 
tions of multiple organs in the same bionic "organ". 

[0132] Almost all biological senses are laid out In a two dimensional array. Touch and taste, olfaction and hearing 
as well. So all of these rhodalities present an image, 2-dimensional image, that can be processed in a similar fashion, 
thus the same CNN universal machine could be used to process them all, individually or collectively. Additionally, 
biologically many modalities overtay In reality and register at the same location - visual, auditory, tactile, thus a CNN 
universal machine that combines sensors for various biological modalities on the same chip and processes them all 
is not so unusual. For many of those senses there are different microsensors that can be placed on the same chip, or 
connected to the same chip, to detect pressure, touch, sight, chemical analysis, etc. What is truly interesting is that 
the CNN universal machine of the present invention, if the time delays to simulate animal perfomnance are omitted, 
can perform all of these functions using the same chip in less time than the biological organism can perfomn those 
same functions. Further, since the entire motor system is triggered at the same point In the brain since visual, auditory, 
tactile stimuli, etc., all converge at the same place, once an interface is developed between the CNN universal bionic 
machine of the present invention and the brain, any of those senses that might have been lost could be replaced, or 
the current senses could even be enhanced. 

[0133] To those skilled in the art to which the invention disclosed herein relates, many changes in construction and 
widely differing embodiments and applications of the Invention will suggest themselves without departing from the spirit 
and scope of the invention. The disclosures and the descriptions herein are purely illustrative and are not intended to 
be in any sense limiting. The actual scope of the invention described herein Is only limited by the scope of the appended 
claims. 
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TABLE I 

Image Processing Templates for 
Figures 9l>-9h 



10 



Function: 



ETOiFTONE 
(Figure 9b) 

Image halftoning: converts the grey-scale input 
image to black and white, still preserving the 
main features of the input. 



15 



-0.030 
-0.080 
-0.130 
-0.080 
-0.030 



-0.080 
-0.359 
-0.604 
-0.359 
-0.080. 



-0.130 
-0.604 
1. 050 
-0.604 
-0.130 



-0.080 
-0.359 
-0.604 
-0.359 
-0.080 



1 

-0.030 
-0.080 
-0.130 
-0.. 080 
-0.030 
J 



20 



25 



0.000 
0.000 
0.320 
0.000 
0.000 



0.000 
0,167 
0.356 
0.167 
0.000 



0. 320 
0.356 
1.000 
0.356 
0.320 



0. 000 
0.167 
0.356 
0.167 
0.000 



0.000 
0.000 
0.320 
0.000 
0.000 



1=0 



30 



35 



40 



AVERAGE 
(Figure 9c) 

Function: spatial averaging of pixel intensities over the 
r=l convolutional window. 



A= 



0 
1 
0 



1 0 

2 1 
1 0 



;B= 



0 
0 
0 



0 
0 
0 



0 
0 
0 



1=0 



45 



50 



55 
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TABLE I (Cont. ) 
image Processing Templates for 
Figures 9to-9h 

Kane : EDGE 
(Figure 9d) 



Function: Black and white edge detector. 



A= 



0 
0 
0 



0 
2 
0 



0 
0 
0 



-0-25 -0.25 -0.25 
-0.25 2 -0.25 
-0.25 -0.25 -0.25 



I=-1.5 



Kame: ON-CEHTER OFF-SDIIROUND 
(Figure 9e) 

Function: Rates a spacial difference over the r=2 convolution 
window. 



0 

0 
0 
0 

o 



O 0 

O 0 

O 0 

0 0 

O 0 



0 
0 

o 

0 



0 

o 
o 
o 



-0.1 -0.2 -0.3 -0.2 -0.1 

-0.2 +0.2 +0.4 +0.2 -0.2 

-0.3 +0.4 +1 +0.4 -0.3 

-0.2 +0.2 +0.4 +0.2 -0.2 

-0.1 -0.2 -0.3 -0.2 -0.1 



1= 
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TABLE I* (Cont. ) 
Image processing Templa-tes tor 
Figiires 91>-9b 



Kame: DXAGGREY 135 
(Figure 9f) 

10 . Function: Detects lines (contours) in a grey-scale image 

being approximately in the north-west, south- 
eastern direction. 



15 



A= 



20 



0 

0 
0 
0 
0 



0 0 0 0 

0 0 0 0 

0 10 0 

0 0 0 0 

O 0 0 0 



b 
b 

a 
a 
a 



b a a a 

b b a a 

b 0 b a 

a b b b 

a a b b 



I=-2.3 



25 



30 





' a 




b 


0.5 






0.5 




-J 


—I 




0.18 ^uij-^uW 


ai8 "m^m 




-0.5 


-0.5 





35 



40 



45 



50 



55 



17 



BNSCXXID: <EP. 



.1 10491 1 A2J_> 



EP1 104 911 A2 



TABLE i (Cont.) 
Image Processing Templates for 
Figures 9b-9h 



10 



Name: DXA66REY 45 
(Figure 9g) 

Function: Detects lines (contours) in a grey-scale image 
being approximately in the south-west, north- 
east orientation. 



15 



A= 



0 

o 

0 
0 
0 



0 0 0 

0 0 0 

10 0 

0 0 0 

0 0 0 



a 
a 
a 
b 
b 



a 
a 
b 
b 
b 



a b 

b b 

0 b 

b a 

a a 



b 
b 
a 
a 
a 



20 



f a 



t b 



25 


0.5 




0.5 






1 


_J 


30 




0.18 ^uij-^ukl 


ai8 V'*'^ 






-0.5 


-0.5 





35 



Function: 



40 



A- 



0 
o 

0 



Name: OR 
(Figure 91i) 

Logical "OR" function of the input and the 
initial state* 



0 o 
3 O 
0 O 



B= 



0 .0 
0 3 
0 0 



o 

0 
0 



1=2 



45 

Clainns 

1. A Method for sensing a selected applied topographic stimulus that is outside the usual range of vertebrate phys- 
iological sensory organs and to present a representation of said stimulus to an output device to present a response 
in a form that is within the usual stimulus range of a selected vertebrate physiological sensory organ, said method 
comprising the steps of: 

a. configuring a cellular neural network (CNN) universal machine (100) to have an array of cells of a selected 
pattern size and shape, and grain density to provide the desired resolution of said topological stimulus, wherein 

55 said grain density is the number of cells per square unit; 

b. selecting a set of sensors (136) for measuring stimulus (1 30) outside the usual range of vertebrate physi- 
ological sensory organs to detect said applied topographic stimulus; 

c. interfacing one of said set of sensors (136) selected in step b. to each cell of said CNN universal machine 
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(1 00) as configured in step a., with each of said set of sensors being responsive to said applied stinnulus and 
developing an output signal compatible with the cell of said CNN universal machine to which it is interfaced 
with said individual signal being in proportion to said stimulus received by said individual sensor; and 
d. programming said CNN universal machine (100) to process signals from each of said CNN cells and to 
5 format a representation of said stimulus to be applied to said output device to present a response in a form 

that is within the usual stimulus range of a selected vertebrate physiological sensory organ. 

2. A method as in claim 1 , wherein said vertebrate is a human being. 

10 3. A method as in claim 1 or 2, wherein said method adaptively models sensory function that is outside the range of 
physiological sensory organs (1 74,1 78). 

4. A method as in any preceding claim, wherein said modelled sensory function exceeds the complexity and speed 
of operation of vertebrate physiological sensory organs (174,178), 

15 

5. A method as in claim 1 , including modelling a plurality of topographic physiological sensory organs (1 74,178), the 
functions of which are to be combined in the same model. 

6. A method as in claim 5, wherein the same topographic physiological sensory organ (174) from two different or- 
20 ganisms is modelled. 

7. A method as in claim 5, Including modelling two different topographic physiological sensory organs (1 74, 1 78) from 
the same organism; and 

selecting a set of stimuli to which said model is to respond from a combination of stimuli to which each of 
^5 said organs respond. 

8. A method as in claim 5 or 6, wherein a processor (102) programs said cellular neural network (CNN) cells to 
simulate perfomnance of a selected second series of steps with those steps selected from a first series of steps 
performed by the same topographic sensory organ of two different living organisms. 

30 

9. A method as in any preceding claim, wherein said functions of said topographic sensory organs (174,178) are 
implemented sequentially 

10. A method as in any preceding claim, further including the step of correlating the responses of said model to said 
35 set of stimuli (130) resulting from a common event. 

11. A method as in any preceding claim, wherein said cellular neural network (CNN) universal machine (100) is pro- 
grammed to simulate a step not perfomied by any known organism. 

"^0 1 2. A sensor to detect a selected applied topographic stimulus that is outside the usual range of vertebrate physiological 
sensory organs and to present a representation of said stimulus in a form that is within the usual stimulus range 
of a selected vertebrate physiological sensory organ, said sensor comprising: 

a cellular neural network (CNN) universal machine (100) having an an-ay of CNN cells of a selected pattern 
size and shape, and grain density to provide a desired resolution of said topographic stimulus, wherein said 
grain density is the number of cells per squire unit; 

a set of sensors (136) for measuring stimulus outside the usual range of said vertebrate physiological sensory 
organs, with one of said sensors interfacing with each CNN cell, each of said sensors disposed to be responsive 
to said topographic stimulus and to each develop a signal in response to said stimulus received by said sensor, 
and then to apply said developed signal to the corresponding one of said CNN cells; 

an output device capable of presenting a response in a form that is within the usual stimulus range of a selected 
vertebrate physiological sensory organ; and 

a processor (1 02) connected to said CNN universal machine (100) and to said output device to sequentially 
program said CNN universal machine and to process signals from each of said CNN cells to format a repre- 
sentation of said stimulus to be applied to said output device. 

13. Apparatus as in claim 12, wherein said vertebrate is a human being. 
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10 



14. Apparatus as In claim 12 or 13, wherein said method adaptively models sensory function that is outside the range 
of vertebrate physiological sensory organs (174,178). 

5. Apparatus as In any of claims 1 2 to 1 4, wherein said modelled sensory function exceeds the complexity and speed 
of operation of vertebrate physiological sensory organs (174,178). 

16. Apparatus as in any of claims 12 to 15, wherein said processor (102) programs said cellular neural network (CNN) 
cells to simulate performance of a selected second series of steps with those steps selected from a first series of 
steps of the same topographic sensory organ (174) of two different living organisms. 



17. Apparatus as in claim 16, wherein said processor (102) stores in said memory (112) program variations for use 
by said processor to alter the program of said CNN cells to simulate a defect in the nomial performance of or an 
interaction between said organs to permit the perfomnance of diagnostic tests on said model to research potential 
corrective actions that may be possible to correct or improve the performance of organs or the interaction between 

15 them. 

18. Apparatus as in any of claims 12 to 17, wherein said processor (102) programs said CNN cells to simulate per- 
formance of a selected second series of steps with those steps selected from a first series of steps of two different 
topographic sensory organs (174,178). 

20 

19. Apparatus as in any of claim 18, wherein said processor (102) stores in said memory (112) program variations for 
use by said processor to alter the program of said CNN cells to simulate a defect in the nonnal perfomnance of or 
an interaction between said organs to permit the performance of diagnostic tests on said model to research potential 
corrective actions that may be possible to correct or improve the performance of said organs (174,178) or the 

^5 interaction between them. 

20. Apparatus as in any of claims 15 to 25, fabricated on a single chip. 

30 
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17 MODELING PARAMETERS 
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Fig. 7a 
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PHOTORECEPTOR INPUT (LAYER 1-136) 
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Fig. 8c 
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